72 research outputs found
Fuzzy bilateral matchmaking in e-marketplaces
We present a novel Fuzzy Description Logic (DL) based approach to
automate matchmaking in e-marketplaces. We model traders’ preferences with
the aid of Fuzzy DLs and, given a request, use utility values computed w.r.t.
Pareto agreements to rank a set of offers. In particular, we introduce an expressive
Fuzzy DL, extended with concrete domains in order to handle numerical, as well
as non numerical features, and to deal with vagueness in buyer/seller preferences.
Hence, agents can express preferences as e.g., I am searching for a passenger car
costing about 22000e yet if the car has a GPS system and more than two-year
warranty I can spend up to 25000e. Noteworthy our matchmaking approach,
among all the possible matches, chooses the mutually beneficial ones
Counterfactual Reasoning for Bias Evaluation and Detection in a Fairness under Unawareness setting
Current AI regulations require discarding sensitive features (e.g., gender,
race, religion) in the algorithm's decision-making process to prevent unfair
outcomes. However, even without sensitive features in the training set,
algorithms can persist in discrimination. Indeed, when sensitive features are
omitted (fairness under unawareness), they could be inferred through non-linear
relations with the so called proxy features. In this work, we propose a way to
reveal the potential hidden bias of a machine learning model that can persist
even when sensitive features are discarded. This study shows that it is
possible to unveil whether the black-box predictor is still biased by
exploiting counterfactual reasoning. In detail, when the predictor provides a
negative classification outcome, our approach first builds counterfactual
examples for a discriminated user category to obtain a positive outcome. Then,
the same counterfactual samples feed an external classifier (that targets a
sensitive feature) that reveals whether the modifications to the user
characteristics needed for a positive outcome moved the individual to the
non-discriminated group. When this occurs, it could be a warning sign for
discriminatory behavior in the decision process. Furthermore, we leverage the
deviation of counterfactuals from the original sample to determine which
features are proxies of specific sensitive information. Our experiments show
that, even if the model is trained without sensitive features, it often suffers
discriminatory biases
Counterfactual Fair Opportunity: Measuring Decision Model Fairness with Counterfactual Reasoning
The increasing application of Artificial Intelligence and Machine Learning
models poses potential risks of unfair behavior and, in light of recent
regulations, has attracted the attention of the research community. Several
researchers focused on seeking new fairness definitions or developing
approaches to identify biased predictions. However, none try to exploit the
counterfactual space to this aim. In that direction, the methodology proposed
in this work aims to unveil unfair model behaviors using counterfactual
reasoning in the case of fairness under unawareness setting. A counterfactual
version of equal opportunity named counterfactual fair opportunity is defined
and two novel metrics that analyze the sensitive information of counterfactual
samples are introduced. Experimental results on three different datasets show
the efficacy of our methodologies and our metrics, disclosing the unfair
behavior of classic machine learning and debiasing models
Aspect-based Sentiment Analysis of Scientific Reviews
Scientific papers are complex and understanding the usefulness of these
papers requires prior knowledge. Peer reviews are comments on a paper provided
by designated experts on that field and hold a substantial amount of
information, not only for the editors and chairs to make the final decision,
but also to judge the potential impact of the paper. In this paper, we propose
to use aspect-based sentiment analysis of scientific reviews to be able to
extract useful information, which correlates well with the accept/reject
decision.
While working on a dataset of close to 8k reviews from ICLR, one of the top
conferences in the field of machine learning, we use an active learning
framework to build a training dataset for aspect prediction, which is further
used to obtain the aspects and sentiments for the entire dataset. We show that
the distribution of aspect-based sentiments obtained from a review is
significantly different for accepted and rejected papers. We use the aspect
sentiments from these reviews to make an intriguing observation, certain
aspects present in a paper and discussed in the review strongly determine the
final recommendation. As a second objective, we quantify the extent of
disagreement among the reviewers refereeing a paper. We also investigate the
extent of disagreement between the reviewers and the chair and find that the
inter-reviewer disagreement may have a link to the disagreement with the chair.
One of the most interesting observations from this study is that reviews, where
the reviewer score and the aspect sentiments extracted from the review text
written by the reviewer are consistent, are also more likely to be concurrent
with the chair's decision.Comment: Accepted in JCDL'2
Reviewing peer review: a quantitative analysis of peer review
In this paper we focus on the analysis of peer reviews and reviewers behavior in a number of different review processes. More specifically, we report on the development, definition and rationale of a theoretical model for peer review processes to support the identification of appropriate metrics to assess the processes main properties. We then apply the proposed model and analysis framework to data sets from conference evaluation processes and we discuss the results implications and their eventual use toward improving the analyzed peer review processes. A number of unexpected results were found, in particular: (1) the low correlation between peer review outcome and impact in time of the accepted contributions and (2) the presence of an high level of randomness in the analyzed peer review processes
Is peer review any good? A quantitative analysis of peer review
In this paper we focus on the analysis of peer reviews and reviewers behavior in conference review processes. We report on the development, definition and rationale of a theoretical model for peer review processes to support the identification of appropriate metrics to assess the processes main properties. We then apply the proposed model and analysis framework to data sets about reviews of conference papers. We discuss in details results, implications and their eventual use toward improving the analyzed peer review processes. Conclusions and plans for future work close the paper
- …